Conversation
|
@kp992 thanks for opening this. Excellent work! It is exciting to see how close we are now. From my review it looks like there a just a few issues and features we need before launch. Bugs:
Features:
|
- Add `markdown: commonmark` to all 24 lecture export configs so ipynb exports produce plain CommonMark markdown cells compatible with vanilla Jupyter Notebook, JupyterLab, and Google Colab - Add build-ipynb.yml workflow that clones QuantEcon/mystmd@myst-to-ipynb, builds from source, exports all ipynb files, and audits for MyST leaks - All 24 notebooks pass audit (0 MyST syntax leaks) Uses QuantEcon/mystmd myst-to-ipynb branch which adds: - CommonMark AST pre-transform (admonitions, math, figures, exercises, etc.) - Identifier/label stripping to prevent (label)= prefixes - Image directive stripping for plain  output - Empty cell filtering and block marker removal
# Conflicts: # .github/workflows/ci.yml
|
Brings the branch up to date with main: the new Polars lecture, the numba lecture rewrite, the pandas_panel switch to remote CSVs, the translation sync workflows, and the jupyter-book 1.x CI changes. The only conflict was .github/workflows/ci.yml, where main and this branch have rewritten the same job for different builders. Resolved in favour of this branch's jupyter-book 2.0 job, adopting main's runner spec (volume=80gb/spot=false) and action bumps (checkout@v7, upload-artifact@v7, actions-netlify@v4) on top of it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…oling Theme: move site.template from the archived QuantEcon/quantecon-theme bundle (main.zip) to a pinned quantecon-theme.mystmd release, v2.2.0 — the current release, which carries the fancy-list rendering that pairs with the fork's fancy-lists parser (qe-v8). Pinned rather than floating so a theme release cannot change the build under us; bump the version in the URL to take a newer one. myst.yml also gets: - project.github corrected to lecture-python-programming (it pointed at lecture-python-programming.myst, which only survives as a rename redirect). The theme derives the notebook-launch repo from this value plus a ".notebooks" suffix, so a stale name means broken launch buttons. - toc brought back in line with _toc.yml: adds autodiff and the new polars lecture, drops the duplicate workspace entry from "Introduction to Python", and renames "The Scientific Libraries" to "Foundations of Scientific Computing". CI: Node 20 -> 24 (the theme's .nvmrc; it runs as a Remix server during `myst build --html`) and the remaining actions to current majors, which also clears the Node 20 runtime deprecation warnings. Adds the JAX install the jupyter-book 1.x workflow on main already does — without it the GPU lectures execute to ModuleNotFoundError cells in the preview, which the build does not fail on. Drops the unused configure-pages step; this job deploys to Netlify, not Pages. mystmd itself already comes from the QuantEcon fork's main branch, built from source with bun, with quantecon/VERSION.yml logged for traceability. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
foo.py, newfile.txt, numbers.txt, output.txt, output2.txt, test_table.csv and us_cities.txt are all written by the lectures themselves during execution (%%file / %%writefile cells in getting_started, python_essentials, python_advanced_features and debugging), and every one of them is matched by .gitignore. They were picked up by the initial mystmd setup commit and do not exist on main. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The preview build was not using QuantEcon/mystmd at all. jupyter-book 2.x is a thin Python shim: jupyter_book/__main__.py execs a compiled mystmd bundle vendored inside its own wheel (jupyter_book/dist/jupyter-book.cjs) and exposes no way to point it at a different CLI. So `jupyter book build` ignored the fork this workflow installs globally and built the site with upstream mystmd, while only the build-ipynb workflow (which calls `myst` directly) ever exercised the fork. The last run shows it plainly: the jupyter-book HTML build emitted 27 x "'output' unknown export output extension: exports/<page>.ipynb", because the per-page ipynb export is a fork feature (myst-to-ipynb, qe-v1) that upstream does not know. The same commit built by the fork's `myst build --ipynb` emitted none and exported 23 clean notebooks. `myst build --html --execute` is the same engine and the same flags — jupyter-book 2.x is mystmd — just our build of it. Dropping the jupyter-book install also removes a second, conflicting mystmd from the image. Workflow renamed to match what it now runs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Refreshed against the current toolchainBranch is up to date with
The HTML build was never using the forkThis is the part worth reading. The evidence is in the last run before the fix — the jupyter-book HTML build emitted 27 × The HTML build now runs Three other things the review turned up
Also: Node 20 → 24 (the theme's Left for a decisionNothing fails the build on a notebook execution error — that is exactly how the JAX breakage stayed invisible. |
{nb-exec-table} directive — execution statistics table (myst-nb parity)
QuantEcon/mystmd#76
|
This is supporting @DrDrij review of quantecon-theme.mystmd |
The GitHub docs URL for "About pull requests" now 301-redirects to https://docs.github.com/en/pull-requests/reference/pull-requests Point directly at the destination so the weekly link checker stops flagging it as a redirect. Reported by the link checker in #587 Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Why this branch builds with QuantEcon/mystmd rather than jupyter-bookRetitled to reflect what the build chain actually does. The short version: jupyter-book cannot be pointed at our fork, so building through it silently discards every QuantEcon feature we depend on. jupyter-book ignores the fork, even when the fork is installedjupyter-book 2.x is a Python shim that execs a mystmd bundle vendored inside its own wheel ( This is not a theoretical concern — it is visible in our own CI history. In the last run that built with The What we would loseThe fork is at What is and is not migrated
Migrating the production chain is deliberately out of scope here and belongs in its own PR: it covers the PDF build (currently Known issue on this branchCold-cache builds currently fail. The execution cache key is This is not caused by mixing build systems — Two things worth doing before diagnosing further: pin the fork clone to the |
Diagnostics for the jax_intro.md kernel death on cold-cache runs (kernel dies with no Python traceback ~66s into execution; numpy.md is collateral). Instruments only -- execution itself is unchanged (default parallelism, no XLA env) so the failure reproduces faithfully: * Pin the QuantEcon/mystmd clone to the exact main SHA the failing runs used (main has not moved since 2026-06-12). A branch name here makes the engine a moving target between runs of the same commit. * Detached 5s sampler logging host RAM, top-RSS processes and GPU memory for the life of the job. * always() step reading the sampler log back plus dmesg, to confirm or rule out the host OOM killer -- a SIGKILL with no traceback is consistent with it but unproven. The cache key only hashes lectures/**/*.md, so this ci.yml-only commit reuses the failing runs' key; no saved cache exists under it (the failed jobs saved none), guaranteeing the cold execution path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fixes the cold-cache kernel death diagnosed in the instrumented run (30806359596). The three JAX lectures execute concurrently on the g4dn runner's single T4; whichever kernel touches the GPU first preallocates JAX's default 75% (11.3GiB of 15.3GiB). The diagnostics showed jax_intro's kernel losing that race, clamping to the ~3.4GiB remainder, and aborting ~40s later when a cell outgrew the pool -- an XLA fatal with no Python traceback, no OOM-killer or segfault records in dmesg, and GPU usage pinned at 3427MiB until the death. The jupyter-book-era green cold run won the same race by seconds -- the engine swap changed execution phasing, not semantics -- and the JB1 production build on main executes sequentially, which is why neither ever hit this. XLA_PYTHON_CLIENT_PREALLOCATE=false switches JAX to on-demand allocation so the concurrent kernels share the GPU. The memory monitor and dmesg diagnostics stay in place; this run doubles as the fix's validation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Hardening after the preallocation-race incident. The cache key now hashes myst_requirements.txt and ci.yml itself alongside the lectures, so any engine SHA bump, JAX bump, XLA env change or requirements change busts the cache and is validated by a cold build automatically -- hashing only the lectures is how the b9cc649 engine swap shipped green without executing a notebook. JAX is pinned to 0.11.0 (the version the 2026-08-03 diagnosis and fix were validated against); a bump is now a deliberate ci.yml edit that itself forces cold validation. Also rewords the diagnosis-era comments: the memory monitor and dmesg read-back stay as standing telemetry. This commit changes the cache key, so its own CI run is cold -- which doubles as a second cold-green validation of the XLA_PYTHON_CLIENT_PREALLOCATE fix under the pinned engine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cold-cache execution failure: diagnosed, fixed, validatedClosing out the "known issue" from my earlier comment. The kernel death was not the engine and not host memory — it was a GPU preallocation race, confirmed by an instrumented run and fixed in two commits. Diagnosis (instrumented run 30806359596)A 5-second GPU/host-memory sampler ran beside the failing build. The GPU timeline told the whole story on one line: 0 → 11,305 MiB → 14,727 → 3,427 → 0 on the g4dn's 15,360 MiB T4. The three JAX lectures ( This also explains the confusing history: the jupyter-book-era cold run went green because Fix and validation
Validation run 30806924581: fully cold (all 25 notebooks executed), zero kernel restarts, Structural fixes, so this class of bug can't ship green againThe engine swap originally shipped green because the execution cache key hashed only One upstream follow-up filed from this: myst reports a dead kernel as a bare |
Experimental setup for building the lectures with the QuantEcon MyST stack. This PR is built on top of #345.
Current toolchain
Both workflows now build with the QuantEcon fork of the MyST CLI and the new theme, so the Netlify preview shows the latest content rendered by the latest tooling.
QuantEcon/mystmdmain, cloned and built from source with bun;quantecon/VERSION.ymlis printed to the log so every build records theqe-vNfork state it ran with (currentlyv1.10.1 (qe-v8)).github/workflows/ci.yml,.github/workflows/build-ipynb.ymlquantecon-theme.mystmdv2.2.0, pinned to the release ziplectures/myst.yml(site.template)myst build --html --execute.github/workflows/ci.ymlmainThe theme moves off the archived
QuantEcon/quantecon-themebundle (which the branch was still pulling as a floatingmain.zip) onto a pinned release of the new theme repo. Bump thevX.Y.Zin thesite.templateURL to take a newer theme; the fork stays onmaindeliberately, so the preview tracks the latest tooling.The build was not actually using the fork
Worth calling out separately, because it had been green and invisible: until this update the HTML preview was built by
jupyter book build, which cannot use the fork. jupyter-book 2.x is a thin Python shim —jupyter_book/__main__.pyexecs a compiled mystmd bundle vendored inside its own wheel (jupyter_book/dist/jupyter-book.cjs) and exposes no hook to point it at another CLI. The globally installed fork was simply ignored, and only thebuild-ipynbworkflow (which callsmystdirectly) ever exercised it.The last run before the fix shows it plainly: the jupyter-book HTML build emitted 27 ×
'output' unknown export output extension: exports/<page>.ipynb, because the per-page ipynb export is a fork feature (myst-to-ipynb,qe-v1) that upstream does not know about. The same commit built by the fork'smyst build --ipynbemitted none and exported 23 clean notebooks. The HTML build now runsmyst build --html --execute— the same engine and the same flags, since jupyter-book 2.x is mystmd, just our build of it.Everything else in this update
main. Brings in the new Polars lecture, the numba rewrite, thepandas_panelswitch to remote CSVs, and the translation-sync workflows. The one conflict wasci.yml, wheremainand this branch have rewritten the same job for different builders — resolved in favour of this branch's job, keepingmain's runner spec and action bumps.myst.ymlTOC re-synced with_toc.yml. It had drifted:autodiffand the newpolarslecture were missing,workspaceappeared twice, and one part title differed. The two files now list exactly the same 27 pages.project.githubcorrected tolecture-python-programming— it pointed atlecture-python-programming.myst, which now only survives as a rename redirect. The theme derives the notebook-launch repo from this value plus a.notebookssuffix, so a stale name means broken launch buttons.ModuleNotFoundErrorcells forjax_intro,autodiffandnumpy_vs_numba_vs_jax, because this job installed onlymyst_requirements.txtwhile the jupyter-book 1.x job onmaininstallsjax[cuda13]separately. Same install, same runner family..nvmrc; it runs as a Remix server duringmyst build --html). This also clears the Node 20 runtime deprecation warnings. The unusedconfigure-pagesstep is gone — this job deploys to Netlify, not Pages.foo.py,newfile.txt,numbers.txt,output.txt,output2.txt,test_table.csv,us_cities.txt). All are written by%%file/%%writefilecells in the lectures themselves, all are matched by.gitignore, and none exist onmain.Still open
maingained a gate for the jupyter-book 1.x path in CI: gate the first jb build so notebook errors cannot pass green #588; the fork's CLI hasmyst build --strictfor this, but it should be turned on only once the remaining build diagnostics are triaged, or it will fail on unrelated warnings.<repo>.notebooks, branchmain, notebooks at the repo root), matching the layout oflecture-python-programming.notebooks. Worth clicking through in the preview to confirm the page location resolves as expected.